Exchange Server 2010 : Planning for Messaging Security

Secure messaging in Exchange 2010 can be separated into three levels: network-based, session (or SMTP)–based, and client-based. It is important to understand at what level you want to implement protection. For example, if you implement network- or session-based security, messages are still not encrypted in a user's mailbox. Only client-based security does this. Alternatively you can also consider implementing security at every level, which definitely never can be reached.

1. Implementing Network-Based Security

Network-based security basically protects the communication on the network layer using protocols such as IPsec or VPN.

IPsec provides a set of extensions to the basic IP protocol and is used to encrypt server-to-server communication. It can be used to tunnel traffic or peer-to-peer to secure all IP communications natively. Because IPsec operates on the transport layer, applications such as Exchange 2010 don't need to be aware of IPsec. You use IPsec normally to secure server-to-server or client-to-server communication. You do not need another encryption method when using IPsec.

Virtual private network (VPN) also operates on the transport layer, and very often uses IPsec as the underlying protocol. VPN is used for site-to-site or client-to-site connections. Both operate on the transport layer, which can be an advantage over application-layer protocols such as Secure MIME (S/MIME), which does not require the application on both ends to know about the protocol.

Because Exchange 2010 by default encrypts its network traffic using TLS and self-signed certificates (if you do not by default roll out server certificates), the requirements for using network-based security for Exchange 2010 are not that important anymore.

Of course, Exchange 2010 also takes advantage of whether you have already implemented network-based secure communication. You don't need to do anything to make Exchange 2010 work; however, to optimize performance, you should consider configuring your connectors accordingly when you have network-based security in place.

Let's assume IPsec is mandatory for all Exchange servers in your organization. You now only need to configure the Receive connectors of your Hub Transport servers and enable Externally Secured on the Authentication tab, as shown in Figure 1. Externally Secured means that the connection is considered secured by a security mechanism that is external to Exchange 2010.

Normally you don't need any other authentication method, but you're able to add only Transport Layer Security (TLS) on top of your network security. However, this will decrease the performance of message transfers because the communication gets encrypted several times. Other options, such as Exchange Server authentication, do not work with Externally Secured. Additionally, you need to configure Exchange Servers on the Permission Groups tab of the Receive connector because this group is used to permit a connection to the server.

2. Planning for Session-Based Security

The TLS protocol is the default protocol used in an Exchange 2010 organization to encrypt server-to-server communication. TLS uses either an available local computer certificate or self-signed certificates that are created during Exchange 2010 setup. This means self-signed certificates provide Exchange 2010 administrators with an easy way to have OWA or other services automatically secured. Self-signed certificates are also used to automatically encrypt messages between Hub Transport and Edge Transport servers to encrypt traffic. They also are used to encrypt traffic between two Edge Transport servers located in different organizations.

If you're planning to implement Exchange 2010 Domain Security to provide secured message paths between Exchange 2010 Edge Transport servers over the Internet, you need real certificates. Self-signed certificates do need extra work when you want to implement Domain Security such as exchanging, installing, and trusting the root Certificate Authority (CA) between both companies.

Domain Security uses TLS with mutual authentication (mutual TLS) to provide session-based authentication and encryption. Standard TLS is used to provide confidentiality by encrypting but not authenticating the communication partners. This is typical of Secure Sockets Layer (SSL), which is the HTTP implementation of TLS.

2.1. Certificates and TLS

Because Hub Transport servers can also use self-signed certificates for their internal communication, what you need to consider here are your Internet-facing Transport servers. On these servers it is recommended that you use an official certificate purchased from a third-party certificate authority (CA) that is well trusted.

The most important requirement on a certificate is to include the following names as SANs:

• The top-level domain for your Exchange organization that your users use in their official e-mail addresses, such as Litware.com

• All FQDNs of the Edge Transport servers

Any certificate that you want to use for TLS requires the following:

• It must be a certificate that was either issued by a trusted party (by importing their root certificate) or by a third-party CA.

• It needs to be installed on the computer's certificate store.

• The certificate must be valid.

• It must be enabled on the Edge Transport server(s) for SMTP service.

On the Edge Transport server you cannot configure certificates in EMC; you have to use the Exchange Management Shell. The cmdlet to enable services for a certificate is Enable-ExchangeCertificate. The cmdlet to verify that the certificate is enabled you can use is Get-ExchangeCertificate |fl, as shown in Figure 2.

To verify that your Edge Transport server is ready to serve mutual TLS requests, you should use the command TELNET <servername> SMTP and verify that when you enter the SMTP command EHLO you see STARTTLS listed. If it is not listed, check your Event Viewer's application log to find out what is wrong with your certificate.

2. Planning for Domain Security

Domain Security in Exchange 2010 is used as a relatively low-cost alternative to S/MIME or other message-encryption solutions. It uses mutual TLS, where each server verifies the identity of the other server by validating the certificate that is provided by the other server. It is an easy way for administrators to manage secured message paths between domains over the Internet.

TLS with mutual authentication differs from TLS in its usual implementation. Typically, when you implement TLS, the client verifies a secure connection to the intended server by validating the server's certificate, which it receives during TLS negotiation. With mutual TLS, each server verifies the connection with the other server by validating a certificate that the other server provides.

Domain Security is manually enabled for every domain by an Exchange organization administrator, so you must coordinate with the communication partner to make it work. It cannot be enabled only on one side, but must be configured in both the sending and receiving organizations.

The high-level steps to implement Domain Security are as follows:

To configure Domain Security, you need to connect to a Hub Transport server (because it is synchronized to the Edge server using EdgeSync) and run the following commands in the EMS:

• To enforce Domain Security on an outbound connection, use the following cmdlet:

  Set-TransportConfig -TLSSendDomainSecureList <DomainList>

  
  

• To enforce domain security on an inbound connection, run this cmdlet:

  Set-TransportConfig -TLSReceiveDomainSecureList <DomainList>

  
  

You need to configure this on a per-domain level. The domain list is not additive—new domains are not automatically added, but replaced. You have to separate the domains using a comma. For example, you can use the following cmdlet to configure outbound domain security for the domains litware.com and fabrikam.com:

Set-TransportConfig -TLSSendDomainSecureList litware.com,fabrikam.com

Your last task is to make sure that the Send connectors and Receive connectors are enabled for Domain Security (Mutual Auth TLS). This is the default configuration, which is enabled if you do not change anything. The Send connector must be configured on the Hub Transport server; the Receive connector must be configured directly on the Edge Transport server.

When you've configured everything correctly, messages from any domain that are on the Domain Secure List should display the Domain Secured icon in Outlook 2007 or later, as seen in Figure 3. If the icon is not displayed, Domain Security might not work correctly and you should do the following to find the issue:

• Check the Event Viewer's application log.

• Check the Queue Viewer in the Exchange Management Console's toolbox.

• Enable Protocol logging on the Internet-facing connectors and take a look into the SMTP log conversation.

3. Implementing Client-Based Security

When you consider client-based security, usually you must also consider Secure Multipurpose Internet Mail Extensions or S/MIME. S/MIME is a standard for public-key encryption and signatures of e-mail messages. Encryption is used to protect the content of a message so that only the intended recipients can read it. Signing a message means that the recipient can verify whether the message has been changed on the way from the sender to the recipient.

S/MIME is a client-based encryption and signing protocol that provides end-to-end security from the sending mailbox to the receiving mailbox. Unlike other encryption protocols that are session-based on the transport layer (such as TLS), the message also remains encrypted and signed within the mailbox. Even administrators cannot decrypt it if their digital certificate does not allow them to do so. Implementing S/MIME offers the following abilities:

• Use digital signatures as a way to prove to your communication partners that the content was not altered.

• Authenticate messages (especially for crucial functions such as when your boss approves your travel requests).

• Encrypt messages to prevent accidental disclosure of the content.

To support S/MIME in your Exchange organization, you must either have a public key infrastructure (PKI) available or your clients need to configure their certificates locally on each client (both their own and the public certificates from the person with whom they want to communicate securely).

If you use Windows PKI, all public certificates of your users are stored in your Active Directory. This allows your users to securely communicate with each other internally. However, if your users often communicate with external partners, you can also make the partner's certificate available in Active Directory. You do this by creating a contact and then publishing the contact's public certificate to it. You can find more information in the "Publish certificates for external contacts in Active Directory" available at http://msexchangeteam.com/archive/2008/04/23/448761.aspx.

Because S/MIME provides end-to-end security, it is important that the e-mail application you use to read and write S/MIME messages meets the following two requirements:

• It must support S/MIME encryption and signatures.

• The digital signature must be configured in the e-mail application.

Outlook Web App running on a Windows system supports S/MIME. If you run OWA on a LINUX system, you do not have the S/MIME feature available and thus you cannot encrypt or decrypt S/MIME messages.